Build Manifest In DevOps Landscape

ABSTRACT

The present invention extends to methods, systems, and computer program products for deriving unified insights ad logs from DevOps CI/CD tools and pipeline data. In general, a data transformer facilitates data normalization and serialization converting raw data across multiple DevOps tools and stores the data into a Data Lake in accordance with a customized schema. A continuous orchestrator sequences, aggregates and contextualizes the logs, providing an intuitive way of troubleshooting issues across a DevOps environment, historical data for compliance and audit purposes, and a build manifest for root cause analysis. The continuous orchestrator also processes the logs and leverages a KPI framework, providing intelligent dashboards across 90+ KPI&#39;s and a plurality of different dimensions (Planning, Development/pipelines, security, quality, operations, productivity and source code) to help customers make smart decisions and do more with less.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 63/164,905 entitled “Unified Insights Across DevOps Landscape” andfiled Mar. 23, 2021, which is hereby incorporated herein by reference inits entirety

TECHNICAL FIELD

The present disclosure relates generally to deriving unified insights,KPIs, and metrics from DevOps data. Aspects include analyzing DevOpsplatform and pipeline data.

BACKGROUND

In general, analytics is the computation analysis of data or statistics.Analytics can be used for discovery, interpretation, and communicationof meaningful data patterns. Analytics also includes applying datapatterns towards effective decision making. In some environments,analytics can be used to quantity performance of software programs,networks, etc. For example, organizations may apply analytics tobusiness data to describe, predict, and improve business performance.Analytics results can be presented via dashboard widgets, rollup bars,counts, reports, etc. In some environments users can customize analyticsresults (at least to some extent).

Software analytics is more specific to the domain of software systemstaking into account source code, static and dynamic characteristics(e.g., software metrics) as well as related processes of theirdevelopment and evolution. It aims at describing, monitoring,predicting, and improving efficiency and effectivity of softwareengineering throughout the software lifecycle, in particular duringsoftware development and software maintenance. The data collection istypically done by mining software repositories, but can also be achievedby collecting user actions or production data. One avenue for using thecollected data is to augment the integrated development environments(IDEs) with data-driven features.

DevOps represents a change in IT culture, focusing on rapid software andinformation technology service delivery through the adoption of agile,lean practices in the context of a system-oriented approach. DevOpsemphasizes people (and culture), and it seeks to improve collaborationbetween operations and development teams. DevOps implementations utilizetechnology—especially automation tools that can leverage an increasinglyprogrammable and dynamic infrastructure from a life cycle perspective.As such, DevOps can be utilized to shorten systems developmentlifecycle, improve the ability to provide continuous software deliveryand help improve quality and security posture.

In general, a platform is an environment in which program code isexecuted. The environment can include hardware, operating system,associated programming interfaces, and other underlying software. Withrespect to DevOps, a DevOps platform provides an environment forcreating and executing DevOps pipelines. A DevOps pipeline can include aset or combination of interconnected tools, also referred to as a“toolchain”, that aids in the delivery, development, and management ofdigital resources (e.g., software) throughout the development lifecycle. DevOps tools can fit into one or more categories supportingDevOps initiatives, such as, for example, plan, create, verify, package,test, release, configure, monitor, validate, version control, andsecurity.

Appropriately configured and executed DevOps platforms and pipelines addsome level of automation to digital resource (e.g., software)development and deployment, increasing agility. However, configuring andexecuting DevOps platforms and pipelines often includes many manualprocesses and can require both significant time (e.g., weeks andpotentially even months) and human resources to complete. For example,DevOps platform configuration can include manually provisioning andallocating various hardware, network, and storage resources from a cloudprovider. DevOps pipeline creation and execution can include manuallymanaging tool interactions with platform resources and between varioustools. In some aspects, human resources are expended to generate “gluecode” (e.g., custom code, custom scripts and manual integration) thatconnects and promotes appropriate interoperation between different toolsin a pipeline/toolchain.

Further, many platform and pipeline environments/providers restrictplatforms to prescribed resource configurations and/or functionality,limit flexibility by preventing customers from selecting their ownchoice of tools, and do not offer out of the box or native integrationto the toolchain to customers.

Analytics on data or statistics from DevOps platforms and pipelines alsohas a number of challenges. Different tools and components can usedifferent and potentially incompatible data types. Further, there may beno clear definition of the relevance of different log data types thatare to be used to discovery, interpret and communicate meaningful datapatterns or quantify performance DevOps performance. Tools can also runon different computing resources in different locations creatingdiversity among log data. Using diverse log data can make it moredifficult to derive meaningful predictions about quality, planning,productivity, security, compliance, and operations. Additionally,different log data may be of interest to different personas,applications, products, projects, sprints, or releases. However,identifying relevant log data across one or more of those dimensions canalso be difficult.

BRIEF DESCRIPTION OF THE DRAWINGS

The specific features, aspects and advantages of the present inventionwill become better understood with regard to the following descriptionand accompanying drawings where:

FIG. 1 illustrates an example block diagram of a computing device.

FIG. 2 illustrates an example computer architecture that facilitatesperforming an analytics operation on standardized analytics data.

FIG. 3 illustrates a flow chart of an example method for performing ananalytics operation on standardized analytics data.

FIG. 4 illustrates an example computer architecture that facilitatesaggregating data in accordance with a build manifest.

FIG. 5 illustrates a flow chart of an example method for aggregatingdata in accordance with a build manifest.

FIG. 6 illustrates an example computer architecture that facilitatesderiving a predictive insight from log data.

FIG. 7 illustrates a flow chart of an example method for deriving apredictive insight from log data.

FIG. 8 illustrates an example computer architecture that facilitatesperforming persona-based analytics operations.

FIG. 9 illustrates a flow chart of an example method for performingpersona-based analytics operations.

FIG. 10 illustrates an example computer architecture that facilitatesdetermining a computing environment posture level.

FIG. 11 illustrates a flow chart of an example method for determining acomputing environment posture level

FIG. 12 illustrates an example computer architecture and flow ofprocessing log data.

FIG. 13 illustrates an example computer architecture and flow ofderiving an insight into computer environment operations.

FIG. 14 illustrates an example computer architecture and flow ofpredicting an insight into computer environment operations.

DETAILED DESCRIPTION

The present invention extends to methods, systems, and computer programproducts for deriving insights from DevOps data, such as, from variousDevOps tools, pipelines, or logs.

Unified insights and logs can be integrated with DevOps tool chainautomation and integrated with declarative pipelines to provide morecomprehensive solutions. DevOps continuous orchestration can provide avariety of benefits with respect to declarative pipelines. DevOpscontinuous orchestration enables users to build the ContinuousIntegration/Continuous Development (CI/CD) declarative pipelines usingdrag and drop techniques. Users can build the pipelines/workflows acrossvarious DevOps stages, tools, and various cloud environments. DevOpsstages can include code commit, software builds, security scans, vaultintegration, approvals, notifications, thresholds and gates, qualitytesting integrations, validation, integrating with change control andmonitoring tools, deployment and providing a view of activity logsacross every stage, etc. Users can also leverage continuous integrationto build pipelines across Software Development Life Cycle (SDLC),Kubernetes, multilanguage pipelines, Infrastructure as a Code (IaC),Salesforce platform, Artificial intelligence, and Machine learningplatform, ServiceNow, and workday platforms.

DevOps Continuous orchestration can also provide a variety of benefitswith respect to unified insights and logs. A data transformer operatesas a data normalization and serialization engine, converting raw dataacross a DevOps environment and tools. The data transformer can storenormalized/serialized data in a data lake in accordance with a platformprovider customized schema. DevOps continuous orchestration can thensequence, aggregate, and contextualize logs and provide an intuitive wayof troubleshooting issues across a DevOps environment. As such, userscan use unified insights and logs to capture historical data forcompliance and audit purposes and provide a build manifest for rootcause analysis. DevOps continuous orchestration can also process logsand leverage a Key Performance Indicator (KPI) framework. DevOpscontinuous integration can provide an intelligent dashboard across anyof number of (e.g., 85+) KPI's and any of a number of different (e.g.,six or more) dimensions (including planning, Development/pipelines,security, quality, operations, and source code) to the customers. Theintelligent dashboard can help users make more informed decisions and domore with less.

Aspects of the invention include data transformation, performinganalytics operations on standardized analytics data, aggregating data inaccordance with build manifests, deriving predictive insights from logdata, performing persona-based analytics operations, and determining acomputing environment posture level.

Analysis can be focused on understanding the past, for example, whathappened and why it happened. Analytics, on the other hand, can befocused on why things happened and what is more likely to happen in thefuture.

Turning to FIG. 1, FIG. 1 illustrates an example block diagram of acomputing device 100. Computing device 100 can be used to performvarious procedures, such as those discussed herein. Computing device 100can function as a server, a client, or any other computing entity.Computing device 100 can perform various communication and data transferfunctions as described herein and can execute one or more applicationprograms, such as the application programs described herein. Computingdevice 100 can be any of a wide variety of computing devices, such as amobile telephone or other mobile device, a desktop computer, a notebookcomputer, a server computer, a handheld computer, tablet computer andthe like.

Computing device 100 includes one or more processor(s) 102, one or morememory device(s) 104, one or more interface(s) 106, one or more massstorage device(s) 108, one or more Input/Output (I/O) device(s) 110, anda display device 130 all of which are coupled to a bus 112. Processor(s)102 include one or more processors or controllers that executeinstructions stored in memory device(s) 104 and/or mass storagedevice(s) 108. Processor(s) 102 may also include various types ofcomputer storage media, such as cache memory.

Memory device(s) 104 include various computer storage media, such asvolatile memory (e.g., random access memory (RAM) 114) and/ornonvolatile memory (e.g., read-only memory (ROM) 116). Memory device(s)104 may also include rewritable ROM, such as Flash memory.

Mass storage device(s) 108 include various computer storage media, suchas magnetic tapes, magnetic disks, optical disks, solid state memory(e.g., Flash memory), and so forth. As depicted in FIG. 1, a particularmass storage device is a hard disk drive 124. Various drives may also beincluded in mass storage device(s) 108 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)108 include removable media 126 and/or non-removable media.

I/O device(s) 110 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 100.Example I/O device(s) 110 include cursor control devices, keyboards,keypads, barcode scanners, microphones, monitors or other displaydevices, speakers, printers, network interface cards, modems, cameras,lenses, radars, CCDs or other image capture devices, and the like.

Display device 130 includes any type of device capable of displayinginformation to one or more users of computing device 100. Examples ofdisplay device 130 include a monitor, display terminal, video projectiondevice, and the like.

Interface(s) 106 include various interfaces that allow computing device100 to interact with other systems, devices, or computing environmentsas well as humans. Example interface(s) 106 can include any number ofdifferent network interfaces 120, such as interfaces to personal areanetworks (PANs), local area networks (LANs), wide area networks (WANs),wireless networks (e.g., near field communication (NFC), Bluetooth,Wi-Fi, etc., networks), and the Internet. Other interfaces include userinterface 118 and peripheral device interface 122.

Bus 112 allows processor(s) 102, memory device(s) 104, interface(s) 106,mass storage device(s) 108, and I/O device(s) 110 to communicate withone another, as well as other devices or components coupled to bus 112.Bus 112 represents one or more of several types of bus structures, suchas a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

Transforming Data

A data transformer can operate as a data normalization and serializationengine that converts raw data (e.g., from a data lake) into a specifiedschema that can subsequently be used for analytics. The data transformerfacilitates converting raw (e.g., log data) data into calculated metricvalues. Query processing time for query data can be reduced from apresentation layer by reducing the number of calculations. The datatransformer can also significantly reduce, if not eliminate, malformeddata from entering analytics systems.

Accordingly, a data transformer can normalize data by, for example,converting raw (e.g., log) data into a standardized (e.g., pipeline)schema. The data transformer can calculate and store metric level datasuch that the query time from the presentation layer can be reduced. Thedata transformer can aggregate and normalize data providing a contextualview to customers. The contextual view helps customers improve the meantime to resolution (MTTR) and operational excellence. Also, helps thecustomers to use the historical data to satisfy compliance and auditrequirements. The data transformer also helps categorize step level dataand group that data in order to create pipeline level data through aconglomeration of step level data.

The data transformer can be built using a loosely coupled architecture.The data transformer can connect to existing customer/user DevOpsecosystems and/or be integrated into managed DevOps environment. Througha customer/user DevOps ecosystem or a managed environment the datatransformer facilitates providing unified insights and logs to helpcustomers to make intelligent decisions. For example, the datatransformer can be included in an analytics system that integratesbusiness objectives and key results (OKRs) with DevOps Key PerformanceIndicators (KPIs). As such, the data transformer can facilitateproviding value stream management metrics for organizations to betterunderstand the value of work that the engineering, developers andoperations are contributing against organization strategy.

A data lake can be a centralized landing point for raw (e.g., log) datacollected across various users, tools, platforms, and pipelines. Usingscalable and dynamic index management, the data lake can segregate rawdata into respective indices which can be searched and queried throughthe data transformer. The data lake can store raw and untouched datapreserved for audit and compliance purposes.

A flexible schema model can wrap stages that run as part of a pipelineinto a standard data model. The flexible schema model allows ananalytics system to more effectively index and store log data. As such,the data can be used in other analytics features, for example, BuildManifest and KPI's. Standardizing a log structure of various stages, theflexible schema model can be utilized to reduce correlation time thequeries that feed KPI's. The flexible schema model also helps link toollevel data to a Pipeline Runs enabling advanced correlation.

The flexible schema model facilitates standardized storage of tool leveldata into a pipeline level correlated schema. The flexible schema modelalso helps prevent malformed data from being ingested reducing thepossibly of breaking an analytics flow.

The flexible schema model can be segregated into at least threeseparated collections, step summary data, pipeline summary data, andexternal collection. Step summary data stores step level pipeline rundata along with any tool specific data associated with a pipeline run.Pipeline summary data is a grouped collection of step summary data for apipeline run. External collection helps standardize the data model fortools that are not part of a pipeline run. The flexible schema modeluses data subnesting with top level identifiers, which helps routequeries to relevant sub nested data.

A real time query processor allows for metrics to be displayed withlimited delay in updating visualizations in an analytics module. Themetrics can relate to various categories including planning,development, security, operations, etc. that span hundreds of KPIs. Thereal time query processor allows users to track metrics across multiplecategories in a real time manner. For example, a user can track progressof their running pipelines and capture tool activity logs (code scans,unit tests, build, testing, vulnerability management, functional andperformance testing, deployment, approvals, etc.) in real-time. Thereal-time query processor allows users to see the results of their dailyoperations as they occur, so that if there any issues or anomalies inthe activity, they can be more quickly identified and resolved. Thereal-time query processor can also provide end to end view of codecommit to code deploy.

The real-time query processor can provide information to usersrelatively quickly. For example, data can be correlated andcontextualized and corresponding visualization processed in essentiallyreal-time. Visualizations can be targeted to developers, managers, andexecutive personas.

FIG. 2 illustrates an example computer architecture 200 that facilitatesperforming an analytics operation on standardized analytics data.

As depicted, computer architecture 200 includes pipeline 221, datanormalizer 201, data lake 202, data aggregator 203, DevOps environment204, and analytics module 207. DevOps environment 204 further includesuser-interface 206.

Pipeline 211 can include a plurality of stages including stages 231A and231B. One or more stages 233 can precede stage 231A. One or more stages234 can be between stages 231A and 2331B. One or more stages 237 canfollow stage 231B. One or more stages 236 can operate in parallel withone or more of stages 231A, 231B, 233, 234, and 237. Each stage caninclude a tool image. For example, stages 231A and 231B include toolimages 232A and 232B respectively.

FIG. 3 illustrates a flow chart of an example method 300 for performingan analytics operation on standardized analytics data. Method 300 willbe described with respect to the components and data in computerarchitecture 200.

Method 300 includes receiving pipeline data including: (1) firstpipeline data in a first format from a first tool running on firstresources through a first API tailored to the first tool and (2) secondpipeline data in a second format from a second tool running on secondresources through a second API tailored to the second tool, the secondformat differing from the first format (301). For example, tool image232A can generate pipeline data 211A and tool image 232B can generatepipeline data 211B. Data normalizer 201 can access pipeline data 211Athrough a first tailored API and can access pipeline data 211B frompipeline 221 through a second tailored API.

In another aspect, data normalizer receives first pipeline dataassociated with a first pipeline and second pipeline data associatedwith a second pipeline. The first pipeline and second pipeline can runserially or in parallel. The first pipeline and the second pipeline canhave a parent child relationship. For example, the first pipeline can bea parent pipeline and the second pipeline a child pipeline of the firstpipeline or vice versa.

Tools (whether in the same pipeline or different pipelines) can run on acombination of cloud resources and/or on-premise resources. Datanormalizer 201 can access pipeline data from tools run on a combinationof cloud resources and/or on-premise resources.

Method 300 includes filtering, validating, normalizing, sequencing,contextualizing, and aggregating first DevOps data from the firstpipeline data (302). For example, data normalizer 201 can filter,validate, normalize, sequence, contextualize, and aggregate DevOps data212A from pipeline data 211A. Method 300 includes storing the firstDevOps data in a data lake (303). For example, data normalizer 201 canstore DevOps data 212A in data lake 202.

Method 300 includes filtering, validating, normalizing, sequencing,contextualizing, and aggregating second DevOps data from the secondpipeline data (304). For example, data normalizer 201 can filter,validate, normalize, sequence, contextualize, and aggregate DevOps data212B from pipeline data 211B. Method 300 includes storing the secondDevOps data in the data lake (305). For example, data normalizer 201 canstore DevOps data 212B in data lake 202.

In one aspect, the first DevOps data is the first pipeline data and thesecond DevOps data is the second pipeline data.

Method 300 includes aggregating the first DevOps data and the secondDevOps data from the data lake into standardized analytics dataincluding normalizing and serializing the first DevOps data and thesecond DevOps data into an analytics format compatible with a clientanalytics environment (306). For example, data aggregator 203 canaggregate DevOps data 212A and DevOps data 212B into standardizedanalytics data 213 (in a format compatible with DevOps environment 204).Aggregation can include normalizing and serializing DevOps data 212A andDevOps data 212B in standardized analytics data 213.

In one aspect, data aggregator 203 converts the first pipeline data andthe second pipeline data in accordance with a standardized pipelineschema. Converting the first pipeline data and the second pipeline datacan include converting the first pipeline data and the second pipelinedata into calculated metric values.

Method 300 includes presenting the standardized analytics data in aclient DevOps environment providing unified insights into clientpipeline operation (307). For example, DevOps environment 204 canpresent standardized analytics data 213 at user-interface 206 providingan insight into operation of pipeline 221. In one aspect, a unifiedinsight is an insight with respect to how one or more pipelines (e.g.,including pipeline 221) are performing against one or more auditrequirements. Presenting standardized analytics data 213 can includepresenting the standardized analytics data by one or more of: product,application, release, project, spring, or family of products at adashboard or log.

Method 300 includes performing a client analytics operation using thestandardized analytics data (308). For example, analytics module 207 canperform a client analytics operation using standardized analytics data213 and generating analytics outcome 214. In one aspect, analyticsmodule 207 uses standardized analytics data 213 to satisfy one or moreof: security, compliance or audit requirements across a DevOpslifecycle. Using standardized analytics data 213 can include crawlingstandardized analytics data 213.

A client analytics operation can include/indicate one or more of:engineering contribution value to a client strategy, developercontribution value to the client strategy, or operations contributionvalue to the client strategy. A client analytics operation can includeevaluating a security posture of one or more tools, such as, forexample, tool image 232A and/or tool image 232B, using KPIs. KPIs camspan multiple dimensions including planning, pipeline, security,quality, operations, source code, and customer success.

In one aspect, a client can use tool chain automation to declarepipeline 221 including tool images 232A and 232B. Data normalizer 201can then access (client) pipeline data 211A and pipeline data 211B fromthe declared pipeline. As depicted, the components of computerarchitecture 200 can be connected to DevOps environment 204. DevOpsenvironment 204 can be a client DevOps environment of the client thatdeclared pipeline 221.

Build Manifest Based Analytics

A build manifest can facilitate unique log analysis. Logs can beprovided each action for automated and manual tasks performed by toolsand system included in pipeline. Build manifests can be automated andstored (e.g., sent to an S3 bucket or in Jira ticket or as an PDF) forlater use. Build manifests can be used for audit and compliance purposesand also for troubleshooting purposes during and after the pipelinedeployment.

Using a build manifest, Development and Operations teams can viewsystem/tool logs for each action without the need to login to any DevOpstools part of the pipeline. Development, Operations and engineeringteams often have to login to multiple tools (15+) on average to fetchand correlate logs. Build manifests eliminate the need of logging intoeach tool or console and enable and empower teams to get the aggregatedlogs and consume them in near real-time via a self-service portal. Thelogs can be received direction from tools/pipelines without alterationallowing the logs to be used for audit and compliance purposes. Usingbuild manifests, users can set up notifications and create tickets inInformation Technology Service Management (ITSM) tools in a proactiveway rather than being reactive. Build Manifests can also facilitatepattern modelling, providing insights to Developers and operations teamon system performance or notify them about the change in pattern on thepipeline.

In general, actionable insights can be derived from any type of raw dataas part of the Pipeline, including Salesforce dot com (SFDC), SoftwareDevelopment Lifecycle (SDLC), Package Deployments, Database Deployments,Infrastructure Provisioning, Infrastructure Deployments, Multi-Languagepipelines, Infrastructure as a Code (IaC), serverless applicationdeployment, Kubernetes application deployment, ArtificialIntelligence/Machine Learning (AI/ML) pipelines, etc. Actionableinsights can provide contextualization and insights to executives tomanagers and developers/operations team on how well their DevOpsecosystem is working right from Planning to production deployment withminimal (and possibly zero) coding.

Actionable insights provide insights to a Developer's code quality,bugs, security issues, TVM's to developers, operations, quality andsecurity team on the fly during pipeline execution. Actionable insightscan be used to create actionable tickets in ITSM for a team to resolveand track as part of a next pipeline execution. Actionable insights canbe trigged based on the thresholds set up by the DevOps team or based onthe pipeline pattern. Actionable insights can be used to predict apipeline using the previous pipeline builds and provide insights andalso provide a value to business and executives on the code quality andbugs that can be expected.

Notifications allow users to receive alerts on the pipeline activity forcomplete, on error, or on all activity in the pipeline. Notificationsalso allow users to build approval notifications for the pipeline forapproval or denial status. Users can set notifications for specific logsthat need to be monitored as part of a pipeline or analytics module toprevent any incidents or failures. Notifications can be sent via email,Slack, HipChat, Microsoft Teams or as an SMS.

Notifications can be configured in API connectors such that a managingsystem can look for any notification configuration and trigger thenotifications based on the information available in a managing systemDB. Notifications can be toggled to off incase notification for part ofpipeline or analytics is not desired or inappropriate. Notifications canbe set to any stage of the pipeline or KPI's or analytics or logs to aspecific group or independent person. Notifications can be set to anyalerts based on the keywords or thresholds in infrastructure/pipelinetools.

Users can set up thresholds or gates for the pipeline or analytics sothat an orchestrator makes decisions and notifies users of thresholdsbreach. Pipeline can be stopped in case there is a breach in thethresholds set up at each stage of the pipeline around build, quality,security, compliance and deployment and application performance KPI's.In case of analytics/KPI's, if the quality or security or Deploy KPI'sbreach a KPI threshold, a notification hub can notify the owners to takecorrective action to stop the breach and can provide remediationinformation. Thresholds can be overridden by having an exceptionapproval stage from a manager or executive.

FIG. 4 illustrates an example computer architecture 400 that facilitatesaggregating data in accordance with a build manifest.

As depicted, computer architecture 400 includes computing resources 421,data normalizer 401, data lake 402, data aggregator 403, and DevOpsenvironment 404. DevOps environment 404 further includes user-interface406. Tools 432A, 432B, and 432C are running on computing resources 421and may be included in one or more pipeline stages. Computing resources421 can be a combination of one or more on-premise resources and/or oneor more cloud resources.

FIG. 5 illustrates a flow chart of an example method 500 for aggregatingdata in accordance with a build manifest. Method 500 will be describedwith respect to the components and data in computer architecture 400.

Method 500 includes receiving log data from a plurality of DevOps toolsrunning on computing resources (501). For example, data normalizer 401can receive log data from two or more of tools 432A, 432B, and 432C. Inone aspect, computing resources 421 include pipeline resources and tools432A, 432B, and 432C run as part of one or more pipelines or pipelinestages on the pipeline resources. Pipelines (or stages thereof) can runserially or in parallel and can include parent/child relationships.Tools (whether in the same pipeline or different pipelines) can run on acombination of cloud resources and/or on-premise resources. Datanormalizer 401 can access pipeline data from tools run on a combinationof cloud resources and/or on-premise resources.

Receiving log data from a plurality of DevOps tools running on computingresources includes receiving first log data from a first tool running onfirst computing resources (502). For example, data normalizer 401 canreceive log data 411A from tool 432A. Receiving log data from aplurality of DevOps tools running on computing resources includesreceiving second log data from a second tool running on second computingresources (503). For example, data normalizer 401 can receive log data411B from tool 432B.

Method 500 includes normalizing the received log data, including firstlog data and the second log data, into a normalized format (504). Forexample, data normalizer 401 can normalize log data 411A and log data411B into normalized log data 412. Method 500 includes storing thereceived log data in a data lake (505). For example, data normalizer 401can store normalized log data 412 in data lake 402.

Method 500 includes accessing a manifest defining log data to beaccessed from each of the plurality of tools, including at least thefirst tool and the second tool (506). For example, data aggregator 403can access manifest 416. Manifest 416 can define log data of interest,for example, log data from a defined plurality of tools, to be accessedfrom data lake 402. Log data of interest can be defined by one or moreof: a pipeline identifier, a build identifier, a product, a sprint, aproject, etc. In some aspects, log data of interest is defined via oneor more filters indicating log data view granularities.

Method 500 includes retrieving building manifest data, including thefirst log data and the second log data, from the data lake responsive toaccessing the manifest (507). For example, data aggregator 403 canaccess build manifest data 414, including log data 411A and log data411B, from data lake 402 in accordance with manifest 416.

Method 500 includes aggregating build manifest data into aggregated logdata (508). For example, data aggregator 403 can aggregate buildmanifest data 414, including log data 411A and 411B, into aggregated logdata 413. In one aspect, data aggregator 403 also sequences aggregatedlog data 413, for example, by pipeline stage of one or more pipelines,into sequenced log data.

In one aspect, data aggregator 403 aggregates log data 411A and log data411B into aggregated log data 403. Data aggregation can include indexinglog data 411A and log data 411B by the one or more of: a pipelineidentifier, a build identifier, a product, a sprint, or a project.

Method 500 includes presenting the aggregated log data and presentingthe build manifest data in a client DevOps environment (509). Forexample, DevOps environment 404 can present aggregated log data 413 andbuild manifest 416 at user-interface 406. If log data was previouslysequenced, DevOps environment 404 can present the sequenced log data atuser-interface 406.

Presenting aggregated log data can include uploading the aggregated logdata to ITSM tools, such as, Jira, Servicenow, etc. Presentingaggregated log data can also include sharing the aggregated log data viaa collaboration tool, such as, Slack, email, MS teams, etc. Presentingaggregated log data can further include sharing the aggregated log datawith or security requesting one or more of: a review or an approval.

Predictive Analytics and Pattern Modeling

Pattern modeling can be used to create models providing predictiveinsights to customers regarding their user patterns, tools patterns,pipelines/operations metadata. Using tracked pattern data, a datatransformer streams metadata and constructs a data streaming modelgiving predictive trends/insights across users, tools, pipelines,operations and planning insights. Insights can be relatively complex andprovide meaningful analysis of a customer's systems across dimensionsincluding security, quality, compliance, operations, etc.

In one aspect, a data pattern modeler can use classification algorithmsand data from various security tools (e.g., static code analysis,dynamic code analysis, Threat and vulnerability scans etc.) to derive apredictive insights model. The predictive insights model can providescore cards right from planning, development, user actions andinfrastructure metadata collected as part of pipeline or tool execution.In another aspect, a data pattern modeler can use data classificationalgorithms and data from various quality categories (e.g., unit testing,functional, API based, Web services, Non functional testing, etc.) toderive a predictive insights model. The predictive insights model canprovide score cards right from planning, development, user actions(manual vs automation) metadata collected as part of pipeline or toolexecution and provide the rate of quality through predictive insights.

Data classification algorithms from the commit can be utilized toprovide metrics including the rate of commit, frequency of failures,success, average build duration. Metrics can be separated by user/byproject/by organization from collected pipeline metadata. Using pipelinemetadata reports can be generated for different personas wherein changescan be implemented to prevent issues or failures.

Insights into DevOps tools rationalization based on the usage,compliance and security issues can be provided to assist withcontrolling DevOps cost around tooling and infrastructure. Insights caninclude recommendations based on planning, Software development,Quality, Security, Operations can be provided to users to improve theirsystem/tools and application performance across releases.Recommendations can also be based on the number of tools being used,number of applications using the tools, number of users using the tool.Recommendations assist with root cause analysis of pipeline issues andpredict failures before they occur. Insights can indicate when and inwhat particular scenario a high or critical error is triggered ininfrastructure or application during the application deployment flow.

Insight scorecards associated with quality, security, compliance,development, etc. can be formulated from log data. Scorecards caninclude:

Quality:

-   -   1. Unit Testing: Tests for individual functions or chunks of        code    -   2. Integration Testing: Tests to ensure new code doesn't damage        existing functionality    -   3. Functional Testing: Tests to ensure code is functioning how        intended

Security:

-   -   1. Static code analysis: Analysis for vulnerabilities in code        that may leave the application susceptible to hacking    -   2. Container Security: Policies and rules that provide access        control to containers to protect infrastructure privacy and        security    -   3. Dynamic code analysis: Analysis for vulnerabilities in code        that may leave the application susceptible to hacking during the        run time    -   4. CIS Benchmarks: CIS benchmark policies will be tracked        against NIST and provide a security scorecard around        infrastructure, cluster and container management.

Compliance:

-   -   1. Incident Management: How well a system would be able to        withstand incident (disaster recovery, mean time to resolve)    -   2. Deployment Efficiency: How frequently deployments occur,        adjusted for issues with deployments

Development:

-   -   1. Development Efficiency: A measure of how quickly code is        developed (adjusted for errors in the code) and committed from        Development environment to QA/Production environments. (Lead        Time for Changes)    -   2. Resource Management: A measure of how efficiently developers        are being used (how much/frequently they commit code    -   3. Provide predictive scorecards and trends on the Manual        (bottleneck) process KPI's/trends around merge request, commit        time, Issues created vs resolved, active contributors, number of        features that can be developed across different application        teams, technologies etc.    -   4. Provide Predictive insights on time taken of maintenance code        checking vs net new code for a new feature

FIG. 6 illustrates an example computer architecture 600 that facilitatesderiving a predictive insight from log data. As depicted, computerarchitecture 600 includes computing users 611, tools 612, pipelines 613,data lake 602, data transformer 603, data pattern identifier 604, datapattern modeler 606, model executor 607, dashboard 608, and storage 641.Users 611 further include 611A, 611 b, and 611C. Tools 612 furtherinclude tools 612A, 612B, and 612C. Pipelines 613 further includepipelines 613A and 613B. One or more of tools 612A, 612B, and 612C canbe included in stages of pipelines 613, such as, stages of pipelines613A, 613B, etc.

FIG. 7 illustrates a flow chart of an example method 700 for deriving apredictive insight from log data. Method 700 will be described withrespect to the components and data in computer architecture 600.

Method 700 includes receiving user log data from one or more users(701). For example, data lake 602 can receive user log data 621 fromusers 611. User log data 621 can include log data from each of users611A, 611B, 611C, etc.

Method 700 includes receiving tool log data from one or more tools(702). For example, data lake 602 can receive user tool log data 621from users 612. User log data 622 can include log data from each ofusers 612A, 612B, 612C, etc. Receiving log data from one or more toolscan include receiving log data from a plurality of tools running onpipeline resources of one or more pipeline stages. Pipeline resourcescan include cloud resources and/or on-premise resources.

Method 700 includes receiving pipeline logs and metadata from one ormore pipelines (703). For example, data lake 602 can receive pipelinelogs and metadata 623 from pipelines 613. Pipeline logs and metadata 623can include log and metadata from each of pipelines 613A, 613B, etc.

In one aspect, various tools 612 run as part of one or more pipelines613 or as part of one or more stages of pipelines 613. The various tools612 can run on pipeline resources. Pipelines 613 (or stages thereof) canrun serially or in parallel and include parent/child relationships.Tools 612 (whether in the same pipeline or different pipelines) can runon a combination of cloud resources and/or on-premise resources. Datalake 602 can access pipeline logs and metadata from tools running on acombination of cloud resources and/or on-premise resources.

Method 700 includes aggregating the user log data, the tool log data,and the pipeline logs and metadata into aggregated log data in a datalake (704). For example, a data aggregator can aggregate user log data621, tool log data 622, and pipeline logs and metadata 623 intoaggregated log data 623 in data lake 602.

Method 700 includes transforming the aggregated log data into analyticslog data into a common data format using a data transformer (705). Forexample, data transformer 603 can transform aggregated log data 623 intoanalytics log data 626. Analytics log data 623 can be in a data formatcommon to and/or compatible with other modules of computer architecture600.

Method 700 includes identifying a data pattern within the analytics logdata with respect to one or more of: a user, a tool, or a pipeline(706). For example, data pattern identifier 604 can identify datapattern 627 from analytics log data 626. The data pattern can relate toone or more of: a user included in users 611, a tool included in tools612, or a pipeline included in pipelines 613.

Method 700 includes generating a model modeling the identified datapattern deriving a predictive insight with respect to the data patternincluding characteristics selected from among: quality, security,compliance, and operations (707). For example, data pattern modeler 606can generate predictive insight model 628 from data pattern 627.Predictive insight model 628 can derive a predictive insight withrespect to data pattern 627 across one or more dimensions selected fromamong: quality, security, compliance, operations, etc.

In one aspect, deriving a predictive insight includes deriving arecommendation to improve pipeline stage operation based on the firsttool, the second tool, and the aggregated log data. In another aspect,deriving a predictive insight includes deriving a root cause of an issuewith a pipeline stage or deriving a prediction of a possible failure ina pipelines stage.

In a further aspect, deriving a predictive insight includes identifyingone or more of: a pipeline stage vulnerability, a pipeline stage bug, ora pipeline stage issue and deriving a recommendation to fix the one ormore of: the pipeline stage vulnerability, the pipeline stage bug, orthe pipeline stage issue.

Method 700 includes saving the model (708). For example, data patternmodeler 606 can save predictive insight model 628 at storage 641.

Method 700 includes providing the predictive insight to a dashboard userinterface in the client analytics environment (709). For example, modelexecutor 607 can execute predictive insight model 628 derivingpredictive insights 629. Model executor 607 can provide predictiveinsights 629 at dashboard 608. Dashboard 608 can be part of a DevOpsenvironment, a quality assurance (QA) environment, a productionenvironment, a pre-production environment, a pipeline stage environment,or a User Acceptance Testing (UAT) environment.

Persona Based Analytics

Personas facilitate user-oriented communication within and between teamsin the organization. Personas help focus design decisions on user goalsand needs rather than on system attributes and features. An objective ofpersonas is to empower users to communicate research data within a team.Utilizing empathy-based data helps users to see the world from thecustomer's perspective. Without personas, organizations risk creating aone-size-fits-all design expected to suit many different users. Theability to manipulate and control every process, event, tool, and so onfrom one place, rather than logging in and out of different platformsand rationalizing data from different sources, can significantlysimplify DevOps process.

An analytics dashboard can be a data and persona driven visualizationpage to bring teams closer together. Based on informed metrics trackingand a (e.g., relatively detailed) customer map, a customer is empoweredto make more informed decisions with respect to design changes,new/updated features, and onboarding strategy. As data volume increases,understanding online analytics numbers and relating them to keyperformance indicators can be challenging. In a single pane of glass,appropriate data can be collected and viewed, including value streamhealth status, performance of tool chain orchestration, tool chainactivities/exceptions, DevOps reports and analytics, traceability, etc.By managing the DevOps lifecycle with value stream metrics, managers cancompare similar value streams across a common set of key performanceindicators (KPIs) to determine DevOps efforts success.

Personas based data crystallizes specific user types, including focusingon core users in the absence of an immediate contact to the end user.KPIs and Metrics can be contextualized from unstructured data intostructured data based on the Customer role and access. Queries can becustomized across a plurality of dimensions including Planning,Pipeline, Quality, Security, Operations. Persona based analytics canalso be used to help uncover patterns of use and trends in behavior thatwould otherwise be masked when lumping together the data for multiplevisitors to a site.

Persona based predictive analytics and data mining, help customersunderstand current patterns data and predict future outcomes. Aligningcontextualized based value metrics to personas helps ensure that revenueexpands right along with the customer's value. Top level executives tendto want to make long-term decisions about business strategy. Personabased predictive analytics can show how business is performing, trendsthat are occurring over time, KPIs and indicators, that assist withexecutive decision making.

Persona based predictive analytics facilitate true autonomy amongindividuals and DevOps team members and Facilitate cross-functionalcollaboration, reducing waste and bottlenecked processes. Persona basedpredictive analytics also facilitate continuous flow across SFDCpipelines (e.g., Continuous Integration (CI) and/or ContinuousDevelopment (CD)), SDLC pipelines, Package Deployments, DatabaseDeployments, Infrastructure Provisioning, Infrastructure Deployments,Multi-Language pipelines, Infrastructure as a Code (IaC), serverlessapplication deployment, Kubernetes application deployment, AI/MLpipelines, etc., including integration, testing, deployment and evenfunding, among others.

Utilizing personas can predict the needs, behavior and possiblereactions of users, that is why they can reduce the use of the usabilityexam. Personas can help product teams figure out the more importantdesign elements. Personas can be used to: (1) combine numbers and humanattributes to create dynamic, more accurate, and continually updateddata-driven persona profiles, (2) turn abstract concept of User into aperson with thoughts and emotions, (3) represent a group of users withsimilar goals and characteristics, (4) help a provider get to users moreclosely to create better experiences.

Connected user personas, stories and maps that rally a team around theneeds and goals of real-world users can be created. Users can bemaintained at the forefront of design decisions, and can be particularlyhelpful in securing buy-in from key stakeholders by putting a face and aname to the needs and pain points your design solutions address. Usingvisual tools helps crate and organize work to gain greater visibility.DevOps reports can be tailored to people at multiple/different levels ofthe organization, who may or may not have in-depth technical knowledge.

There is a number of beneficial uses associated with persona-basedanalytics, including:

-   -   Take the data, segment it by the personas mentioned above, and        analyze the differences and similarities.    -   Aggregated KPIs across Continuous Integration and Continuous        development toolchain for a single view.    -   Measure Approval gates, accountable owners and updates for        faster resolution and improved auditing and compliance.    -   Identify, detect, and eventually predict trends and behavior by        persona Understand different groups of user interactions.    -   Gather quantifiable data—Identify, detect, and eventually        predict trends and behavior by users.    -   Normalize the technology stack—Help users to normalize their        tech stacks by eliminating redundant systems, perhaps        refactoring applications to work on a smaller set of operating        systems.    -   Help customers to Identify Homegrown tools and services that        don't follow any common industry standards and lack common        interfaces.    -   More flexibility for development staff to work on different        applications, services or components thereby minimizing the        technical ramp up time.    -   Reduced surface area for security vulnerabilities    -   Individuals can make changes without significant wait times    -   Conducting a security review for all major features while        ensuring that the security review process does not slow down        development.    -   Testing security requirements as a part of the automated testing        process.

Persona-based analytics can be designed across multiple dimensions(including Planning, Development, Security, Operations) of DevOpsevolution. Practices in each dimension help customers achieve successand progress to the next phase of DevOps journey. Customers can viewpersonal based charts for the installed tools and services.

Design can implement the following principles:

-   -   Flexible—Flexible enough to perform end-end orchestration using        Customized filters like Projects, Features, Releases,        Contextualized Key Performance Indicators on any given time.        Custom driven real-time queries which can be exported can speed        up in identifying warnings and errors for specific tools, users.    -   Predict and Prevent—AI driven intelligent charts can help find        anomalies and recommend fixes and suggestions providing clear        visibility for accountable owners.    -   Inbuilt Security—Inbuilt security and quality gates that can        trigger notifications for preventive measures. Details could        also be identified by applying, or removing, certain filters.

In one aspect, a Modular, loosely coupled, microservices andcontainer-based architecture is used. Also, a data transformer, realtime query processor, and DB schema can be used (e.g., as described inthe “Data Transformation” section of this application).

FIG. 8 illustrates an example computer architecture 800 that facilitatesperforming persona-based analytics operations.

As depicted, computer architecture 800 includes computing resources 821,data lake 802, request processor 803, dashboard formulator 804, anduser-interface 806. Tools 822A and 822B and pipelines 823A and 823B arerunning on computing resources 821. Tools 822A and 822B may be includedin one or more pipeline stages.

FIG. 9 illustrates a flow chart of an example method 900 for performingpersona-based analytics operations. Method 900 will be described withrespect to the components and data in computer architecture 800.

Method 900 includes accessing raw log data from a log aggregator, thelog data associated with a plurality of tools running on computingresources (901). Accessing raw log data from a log aggregator includesreceiving first log data associated with at least one of: a first toolrunning on first computing resources or a first pipeline running on thefirst computing resources (902). Accessing raw log data from a logaggregator includes receiving second log data associated with at lastone of: a second tool running on second computing resources or a secondpipeline running on the second computing resources (903). Method 900includes storing the accessed raw log data in a data lake (904). Forexample, a log aggregator can receive raw log data 811A and raw log data811B from tools and/or pipelines running on computing resources 821. Thelog aggregator can store raw log data 811A and raw log data 811B in datalake 802.

Tools running on computing resources 821 can be included in differentpipelines and/or different pipeline stages. Data lake 802 can receivedata associated with different tools and/or pipelines. Differentpipelines (or stages thereof) can run serially or in parallel. Pipelinescan also have parent/child relationships. Tools (whether in the samepipeline or different pipelines) can run on a combination of cloudresources and/or on-premise resources. Data lake 802 can storeaggregated log data from tools run on a combination of cloud resourcesand/or on-premise resources.

Method 900 includes receiving a user request for a specific type of rawlog data from a user (905). For example, request processor 803 canreceive request 814, including properties 815 and data type 816. In oneaspect, data type 816 indicates a data type from among: planning,pipeline, quality, security, operations, source code, or customersuccess.

Method 900 includes determining request properties including one or moreof: a persona, an application, a product, a project, a sprint, or arelease associated with the request (906). For example, requestprocessor 803 can determine properties 815 of request 814. Determiningproperties 815 can include determining a persona selected from amongdeveloper, manager, or executive.

Method 900 includes identifying performance indicators (e.g., KPIs) thatboth: (a) correspond to the request properties and (b) are associatedwith the specified type of raw log data from a pre-defined list ofperformance indicators (907). For example, request process 803 canidentifier KPIs 817, including KPI 817A and KPI 817B, that bothcorrespond to properties 815 and are associated with data type 816. KPIscan span the dimensions of planning, pipeline, security, quality,operations, source code, and customer success.

In one aspect, identifying performance indicators includes receiving auser selection of one or more Key Performance Indicators (KPIs), forexample, via a drag and drop (interface) model.

Method 900 includes submitting log data query representative of theperformance indicators to the data lake (908). For example, requestprocessor 803 can send query 812 to data lake 802. Query 812 can be aquery representative of KPIs 817.

Method 900 includes receiving responsive raw log data, including atleast a portion of the first log data and a portion of the second logdata, from the data lake responsive to the query (909). For example,responsive to query 812, request processor 803 can receive raw log data813 form data lake 802. Raw log data 813 can include at least a portionof raw lag data 811A and at least a portion of raw log data 811B.Request processor 803 can send raw log data 813 to dashboard formulator804. Dashboard formulator 804 can receive raw log data 813 from requestprocessor 803.

Method 900 includes formulating a custom dashboard contextualizing andcorrelating the responsive raw log data for the user (910). For example,dashboard formulator 804 can formulate custom dashboard 818 from raw logdata 813. Custom dashboard 818 can contextualize and correlate raw logdata 813 to the user.

Method 900 includes presenting the custom dashboard to the user (911).For example, dashboard formulator 804 can present custom dashboard 818at user-interface 806.

In one aspect, custom dashboard 818 is compared to a previouslyformulated custom dashboard associated with data type 816. Based on thecomparison, dashboard formulator 804 (or some other module) can identify(1) a trend in operation of at least one of: a tool and/or pipelinerunning on computing resources 821 or (2) information related tosecurity, quality, or operational posture of at least one of: a tooland/or pipeline running on computing resources 821.

Deriving Computing Environment Posture

Unified insights and logs can help customers better understand theircurrent DevOps ecosystem security and compliance score cardsautomatically using various techniques. Using a security and complianceframework, customers can automatically validate controls against thecurrent environment and get a report with score card and the currentposture level (basic, intermediate and advanced). Results can begenerated using an automatic process. Controls can be checked againstcustomer DevOps ecosystem using the pipeline workflows, logs fromvarious tools and pipelines, build blueprints, API connectors and toolregistry. Controls can be validated in multiple various dimensions andcapabilities including developers, operations, quality and security).Based on validation output, scorecard against the customer DevOpsenvironment can be generated.

Various different controls can used, including any of the following:authentication, authorization, identity, Infrastructure securitycontrols, encryption, configuration changes, log management, userprivileges, vulnerabilities, certificates, clear text passwords,Inventory of tools/assets, data recovery, monitoring and alerting,security access to the environment, static code analysis, quality andsecurity gates and thresholds and incident response capabilities.

FIG. 10 illustrates an example computer architecture 1000 thatfacilitates determining a computing environment posture level.

As depicted, computer architecture 1000 includes computing resources1021, data lake 1001, data normalizer 1002, validator 1003, anduser-interface 1004. Tools 1022A and 1022B and pipelines 1023A and 1023Bare running on computing resources 1021. Tools 1022A and 1022B may beincluded in one or more pipeline stages.

FIG. 11 illustrates a flow chart of an example method 1100 fordetermining a computing environment posture level. Method 1100 will bedescribed with respect to the components and data in computerarchitecture 1000.

Method 1100 includes receiving log data from a computing environment(1101). Receiving log data from a computing environment includesreceiving first log data from at least one of: a first tool running onfirst computing resources or a first pipeline running on the firstcomputing resources (1102). Receiving log data from a computingenvironment includes receiving second log data from at least one of: asecond tool running on second computing resources or a second pipelinerunning on the second computing resources (1103). Method 1100 includesstoring the received log data in a data lake (1104). For example, a logaggregator can receive raw log data 1011A and raw log data 1011B fromtools and/or pipelines running on computing resources 1021. The logaggregator can store raw log data 1011A and raw log data 1011B in datalake 1001.

Tools running on computing resources 1021 can be included in differentpipelines and/or pipelines stages. Data lake 1001 can receive dataassociated with different tools and/or pipelines. Different pipelines(or stages thereof) can run serially or in parallel. Pipelines can alsohave parent/child relationships. Tools (whether in the same pipeline ordifferent pipelines) can run on a combination of cloud resources and/oron-premise resources. Data lake 1001 can access pipeline data from toolsrun on a combination of cloud resources and/or on-premise resources.

Method 1100 includes normalizing, sequencing, and contextualizing thefirst log data and the second log data from the data lake intoaggregated log data (1105). For example, data normalizer 1002 can accessraw log data 1011A and raw log data 1011B from data lake 1001. Datanormalizer 1002 can normalize, sequence, and contextualize raw log data1011A and raw log data 1011B into aggregated log data 1013. Datanormalizer can send aggregated log data 1013 to validator 1003.Validator 1003 can receive aggregated log data 1013 from data normalizer1002.

Method 1100 includes validating security, quality, and compliancecontrols against the aggregated log data determining a security,quality, and compliance posture level of the computing environment(1106). For example, validator 1003 can validate security, quality, andcompliance controls against aggregated log data 1013 determining posturelevel 1014 of computing environment 1012.

In one aspect, 1106 includes checking the security, quality, andcompliance controls against a customer DevOps ecosystem (e.g., 204 or404) using one or more of: pipeline workflows, aggregated log data 1013,build manifests (e.g., 416), API connectors, or a tool registry.

In another aspect, 1106 includes validating one or more security,quality, and compliance controls selected from among controls related toone or more of: authentication, authorization, identity, infrastructuresecurity controls, encryption, configuration changes, log management,user privileges, vulnerabilities, certificates, clear text passwords,inventory of tools/assets, data recovery, monitoring and alerting,security access to the environment, static code analysis, dynamic codeanalysis, container scans, quality gates, security gates, thresholds,and incident response capabilities.

In a further aspect, 1106 includes determining computing environmentcompliance with one or more government regulations across security,compliance, and quality dimensions. Government regulations can includeSOX, PCI, HIPPA, FTC, or FedRAMP

In an additional aspect, 1106 includes determining a maturity level ofone or more tools (1022A, 1022B, etc.) and/or one or more pipelines(1023A, 1023B, etc.) from among: beginner, intermediate, or advanced.

In another further aspect, 1106 includes: (1) identifying a gap in oneor more tools (1022A, 1022B, etc.) and/or one or more pipelines (1023A,1023B, etc.) or (2) identifying an improvement that can be made to oneor more tools (1022A, 1022B, etc.) and/or one or more pipelines (1023A,1023B, etc.)

Method 1100 includes presenting the security, quality, and complianceposture level in a security and compliance scorecard in a one of: aDevOps environment, a quality assurance (QA) environment, a productionenvironment, a pre-production environment, a pipeline stage environment,or a UAT environment (1107). For example, validator 1003 can presentposture level 1014 at user-interface 1004. Presenting the security,quality, and compliance posture can include one or more of: (1)presenting a maturity level (e.g., beginner, intermediate, or advanced)of one or more tools (1022A, 1022B, etc.) and/or one or more pipelines(1023A, 1023B, etc.), (2) presenting an improvement that can be made toone or more tools (1022A, 1022B, etc.) and/or one or more pipelines(1023A, 1023B, etc.), or (3) presenting the gap in one or more tools(1022A, 1022B, etc.) and/or one or more pipelines (1023A, 1023B, etc.)

Other Aspects

FIG. 12 illustrates an example architecture and flow 1200 of processinglog data.

Analytics engine 1201 includes API 1211, middleware 1212, micro service1213, collaboration tools notification 1214, email notification 1216,ITSM tick creation 1217, validate 1218, data lake 1219, data transformer1221, DB schema update 1222, micro service 1223, middleware 1224, API1226, and filters 1227. In general, analytics engine 1201 can handleunified insights in DevSecOps space across planning, source code,pipeline, quality, security, and operations.

Analytics engine 1201 can be connected to any DevOps tools that startsfrom source code management, Build, Testing, Security, Deploy, Planningtools and IT service management tools for unified insights. For example,in flow 1200, analytics engine 1201 is connected to tools 1202. Tools1202 include planning tools 1202A, development tools 1202B, static codeanalysis 1202C, testing tools 1202D, dynamic scan 1202E, container scantools 1202F, deploy tools 1202G, operations tools 1202H, ITSM tools12021, and AL Ops/PM 1202J. Tools 1202 are connected to analytics engine1201. Tools 1202 can emit log data and/or metadata to analytics engine1201.

Analytics engine 1201 is further connected to user interface 1206. Userinterface 1206 includes build manifest/build blue print 1231, logs 1232,actionable insights/remediation 1233.

Lightweight Directory Access Protocol (LDAP) integration can provideRole based access for users who can be classified as, engineer, Qualityengineers, Security engineers, Managers, Executives etc. For example,users can log in using Lightweight LDAP, Security Assertion MarkupLanguage (SAML), customer Oauth, etc. 1203. LDAP Auth Service 1204 canvalidate users who login with the right LDAP group and provide them theaccess they are intended to access, such as, for example, developer1241, manager 1242, or executive 1243.

In general, Application Programming Interfaces (APIs) 1211 and 1226connect with tools and internal services, for example, to post messagesto or consume data from another service or service layer. For example,API 1211 connects tools 1202 and analytics engine 1201. Log data and/ormetadata emitted from tools 1202 can be received at API 1211. Similarly,API 1226 connects user interface 1206 and analytics engine 1201.Analytics engine 1201 can used API 1226 to send logs, insights,notifications, etc. to user interface 1206

Middleware 1212 (e.g., Kafka) can queue messages and post messagesmultiple times or post messages to multiple services at the same time.Micro service 1213 (a java based service) can be used to connect a nodelayer to end point tools and take action accordingly based on a nodelayer call.

User notifications can be sent using any of a variety of collaborationtools 1214 including email 1216, Slack, MS Teams, pager duty Jira tools,ITSM ticket creation 1217, etc. to create a ticket or update the userbased on the threshold set up by the portal user for a tool or pipelineor unified insights KPI's. Notifications can be set by the user for apipeline or tool or insights based on the metadata or a threshold.Microservice 1213 can interoperation with ITSM tools or incidentmanagement systems to create tickets

Validate 1218 can validate log/metadata types are in an appropriateformat before pushing log/metadata to data lake 1219. Data lake 1219 isa repository for raw data collected from tools and platforms (e.g., anyof tools 1202). Using scalable and dynamic index management, data lake1219 can segregate data into respective indices which can be searchedand queried via data transformer 1221. Data lake 1219 can preserve rawdata for audit and compliance purposes.

Data transformer 1221 can normalize and serialize raw data from datalake 1219 into a schema used for analytics. Data transformer 1221 canconvert raw into calculated metric values improving query processingtime from a presentation layer and reducing the need for repeatedcalculations. Data transformer 1221 can also help present malformed datafrom entering databases facilitating more efficient and effective end toend analytics flow.

DB (database) schema update 1222 can include schema for variouscategories of tools including Software Configuration Management (SCM),build, deploy, Quality, Security, Planning, operations and ITSM tools.Unified insights can be offered across all of the various categories andcan be filtered based on different tool vendors with various filteroptions in the UI.

Micro service 1223 (a java based service) can be used to connect a nodelayer to end point tools and take action accordingly based on a nodelayer call. Middleware 1224 (e.g., Kafka) can queue messages and postmessages multiple times or post messages to multiple services at thesame time.

Filters 1227 can be used to filter data based on tools, persona, timefilter, Feature, release, organization level, etc. in UI for users tofilter results based on their needs.

User interface 1206 can be used for Tool chain automation, Pipeline andUnified insights, notifications, etc. In one aspect, user interface 1206is built on React JS. User interface 1206 can also present any of buildmanifest/build blue print 1231, logs 1232, actionableinsights/remediation 1233.

Build manifest/build blue print 1231 can provide unique log analysis,essentially providing logs with each activity for automated and manualtasks from tools and used as part of a pipeline. Build manifest/buildblue print 1231 can be automated and sent to S3 bucket or Jira ticket oras an PDF which can be later used for audit and compliance purposes andalso for troubleshooting purposes during and after the deployment.

Logs 1232 can include tools & pipeline metadata that is parsed throughand/or generated by a pipeline or tool. Logs 1232 can be shown at userinterface 1206 to any user having access (e.g., based on Role-BasedAccess Control (RBAC)), for example, developer 1241, manager 1242,executive 1243, etc. Logs 1232 can used for trouble shooting andcommunication

Actionable insights/remediation 1233 take any type of raw data as partof the Pipeline (SFDC, SDLC, Package deployments, DB deployments orInfrastructure provisioning, Infrastructure deployments, etc.) andprovide contextualization and insights to executives to managers anddevelopers/operations team on how well their DevOps ecosystem is workingright from Planning to production deployment with zero coding.

FIG. 13 illustrates an example architecture and flow 1300 of deriving aninsight into computer environment operations.

Analytics engine 1301 includes API 1311, middleware 1312, actionableinsights/remediation 1314, logs 1316, Build manifest/build blue print1317, validate 1319, data lake 1219, data transformer 1321, DB schemaupdate 1322, middleware 1324, real time query processor 1351, patternmodeling 1352, predictive analytics processor 1353, andproject/product/tools filter 1354. In general, analytics engine 1301 canhandle unified insights in DevSecOps space across planning, source code,pipeline, quality, security, and operations.

Analytics engine 1301 can be connected to any DevOps tools that startsfrom source code management, Build, Testing, Security, Deploy, Planningtools and IT service management tools for unified insights. For example,in flow 1300, analytics engine 1301 is connected to tools 1302. Tools1302 include planning tools 1302A, development tools 1302B, static codeanalysis 1302C, testing tools 1302D, dynamic scan 1302E, container scantools 1302F, deploy tools 1302G, operations tools 1302H, ITSM tools13021, and AL Ops/PM 1302J. Tools 1302 are connected to analytics engine1301. Tools 1302 can emit log data and/or metadata to analytics engine1301.

Analytics engine 1301 is further connected to user interface 1306,notifications/thresholds 136, and create ticket for ops 1357. Userinterface 1206 includes real time KPIs 1331, predictive KPIs analytics1332, and customer KPIs 1333.

Lightweight Directory Access Protocol (LDAP) integration can provideRole based access for users who can be classified as, engineer, Qualityengineers, Security engineers, Managers, Executives etc. For example,users can log in using Lightweight LDAP, Security Assertion MarkupLanguage (SAML), customer Oauth, etc. 1303. LDAP Auth Service 1304 canvalidate users who login with the right LDAP group and provide them theaccess they are intended to access, such as, for example, developer1341, manager 1342, or executive 1243.

In general, Application Programming Interface (APIs) 1311 connects withtools and internal services, for example, to post messages to or consumedata from another service or service layer. For example, API 1311connects tools 1302 and analytics engine 1301. Log data and/or metadataemitted from tools 1302 can be received at API 1311.

Middleware 1312 (e.g., Kafka) can queue messages and post messagesmultiple times or post messages to multiple services at the same time.

Validate 1318 can validate log/metadata types are in an appropriateformat before pushing log/metadata to data lake 1319. Data lake 1319 isa repository for raw data collected from tools and platforms (e.g., anyof tools 1302). Using scalable and dynamic index management, data lake1319 can segregate data into respective indices which can be searchedand queried via data transformer 1321. Data lake 1319 can preserve rawdata for audit and compliance purposes.

Data transformer 1321 can normalize and serialize raw data from datalake 1319 into a schema used for analytics. Data transformer 1321 canconvert raw into calculated metric values improving query processingtime from a presentation layer and reducing the need for repeatedcalculations. Data transformer 1321 can also help present malformed datafrom entering databases facilitating more efficient and effective end toend analytics flow.

DB schema update 1322 can include schema for various categories of toolsincluding Software Configuration Management (SCM), build, deploy,Quality, Security, Planning, operations and ITSM tools. Unified insightscan be offered across all of the various categories and can be filteredbased on different tool vendors with various filter options in the UI.

Middleware 1324 (e.g., Kafka) can queue messages and post messagesmultiple times or post messages to multiple services at the same time.

Actionable insights/remediation 1233 take any type of raw data as partof the Pipeline (SFDC, SDLC, Package deployments, DB (database)deployments or Infrastructure provisioning, Infrastructure deployments,etc.) and provide contextualization and insights to executives tomanagers and developers/operations team on how well their DevOpsecosystem is working right from Planning to production deployment withzero coding.

Logs 1316 can include tools & pipeline metadata that is parsed throughand/or generated by a pipeline or tool. Logs 1316 can be utilized byanalytics engine 1301 for any user having access (e.g., based onRole-Based Access Control (RBAC)), for example, developer 1241, manager1242, executive 1243, etc. Logs 1316 can used for trouble shooting andcommunication

Build manifest/build blueprint 1317 can provide unique log analysis,essentially providing logs with each step for automated and manual tasksfrom the tools and system that is used as part of the pipeline. Thesebuild manifests can be automated and sent to S3 bucket or Jira ticket oras an PDF which can be later used for audit and compliance purposes andalso for troubleshooting purposes during and after the deployment.

Real time query processor 1351 allows for metrics to be displayed atuser interface 1306 with minimal delay in updating visualizations.Metrics can relate to distinct dimensions (Planning, Development,Security, Operations) that span across numerous (e.g., 100+) KPI's. Realtime query processor 1351 allows users to track the metrics across thesedimensions in a real time manner. (such as tracking the progress oftheir pipelines as they are running. In addition, logs for tool activityshould be captured in real-time as well). Real time query processor 1351allows users to see the results of their daily operations as they occur,so that if there any issues or anomalies in the activity, they can beidentified and resolved.

Pattern modelling 1352 is used to create models to provide predictiveinsights to customers regarding their user patterns, tools patterns,pipelines/operations metadata, etc. Using log/metadata emitted fromtools 1302, pattern modelling 1352 can construct data streaming modelsto give predictive trends/insights across users, tools, pipelines,operations, and planning insights. Pattern modelling 1352 can derivemodels that provide predictive analytics to customers. Model derivationcan be relatively complex generating models providing more meaningfulanalysis/analytics of user systems.

Predictive analytics processor 1353 provides insights on when and inwhat particular scenario a (e.g., high or critical) error is triggeredin an infrastructure or application during the application deploymentflow.

Filters 1354 can be used to filter data based on tools, persona, timefilter, Feature, release, organization level, etc. in UI for users tofilter results based on their needs.

Real Time KPIs 1331 can be output from real time query processor 1351.

Predictive KPI's analytics 1332 can be output from predictive analyticsprocessor 1353 executing a model derived by pattern modeler 1352.

Custom KPIs 1333 can be customer created KPI's relevant totools/pipelines used by the customer.

Notifications/thresholds 1356 allows users to receive alerts on thepipeline activity for complete, on error, or on all activity in thepipeline. Notifications/thresholds 1356 also allow users to buildapproval notifications for the pipeline for approval or denial status.Users can set notifications for specific logs that need to be monitoredas part of a pipeline or analytics module to help prevent any incidentsor failures. Notifications can be sent via email, Slack, HipChat,Microsoft Teams or as an SMS.

An ITSM microservice can create 1357 incident management or serviceticket to Information technology Service Management (ITSM) tools likeJira or Service now.

FIG. 14 illustrates an example architecture and flow 1400 of predictingan insight into computer environment operations.

Pipelines 1401 can include a plurality of DevOps tools stitched togetherfrom commit code to deploy. Pipeline 1401 can also be used to processany stage-by-stage process set up by the user for orchestrationpurposes. Pipelines 1401 can include declarative pipelines.

Tools 1402 can include Dev, QA, Security, Deploy, ITSM and collaborationtools connected for Tool chain automation, pipeline or unified insights.

Users 1403 include persona's with access to unified insights via UI orthrough API layer, who have authenticated, and are authorized to consumedata or pipeline or tool chain automation.

Log aggregator 1404 consumes any metadata from pipelines 1401, tools1402, users 1403, etc. including any of: Dev, QA, Security, Deploy,Planning and ITSM, collaboration tools. Consumed data can used toprovide unified insights after data transformation.

Kafka 1406 is a middle ware service used to queue messages. Kafka 1406can post a message at multiple times or to multiple services at the sametime.

Data transformer 1407 can normalize and serialize raw data from a datalake into a schema used for analytics. Data transformer 1407 can convertraw into calculated metric values improving query processing time from apresentation layer and reducing the need for repeated calculations. Datatransformer 1407 can also help present malformed data from enteringdatabases facilitating more efficient and effective end to end analyticsflow.

Customer metadata DB 1406 can store any customer data for registrationlike First name, Last name, email Address, Location, phone number,cloud, etc. A list of tools used by a customer can also be consideredcustomer meta data. A customer can be included in users 1403.

Tools/pipeline platform DB 1409 can store metadata for any of pipelines1401 and/or tools 1402.

Node 1411 can be Node JS, a Java script API layer, used as anorchestration layer for tool chain automation, pipeline and unifiedinsights.

Predictive service 1412 can provide insights on when and in whatparticular scenario a (e.g., high or critical) error is triggered ininfrastructure or application during the application deployment flow.

Save model 1413 can be used in data modelling whenever a new pattern isidentified and is to be used for actionable or predictive insights toinform the user to identify/mitigate failures.

Predictive KPI processor 141 can provide insights on when and in whatparticular scenario a (e.g., high or critical) error is triggered ininfrastructure or application during the application deployment flow.

Operas dashboard UI 1416 can include any of the functionality describedwith respect to user interfaces 1206 and/or 1306.

Engineers/Managers/Executives can be different persona's set up foraccessing pipeline, tool chain automation, and insights. Users/Customerscan create multiple persona's apart from the default persona's(Engineers/Managers/Executives).

In the above disclosure, reference has been made to the accompanyingdrawings, which form a part hereof, and in which is shown by way ofillustration specific implementations in which the disclosure may bepracticed. It is understood that other implementations may be utilizedand structural changes may be made without departing from the scope ofthe present disclosure. References in the specification to “oneembodiment,” “an embodiment,” “an example embodiment,” etc., indicatethat the embodiment described may include a particular feature,structure, or characteristic, but every embodiment may not necessarilyinclude the particular feature, structure, or characteristic. Moreover,such phrases are not necessarily referring to the same embodiment.Further, when a particular feature, structure, or characteristic isdescribed in connection with an embodiment, it is submitted that it iswithin the knowledge of one skilled in the art to affect such feature,structure, or characteristic in connection with other embodimentswhether or not explicitly described.

Implementations can comprise or utilize a special purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more computer and/or hardware processors (including anyof Central Processing Units (CPUs), and/or Graphical Processing Units(GPUs), general-purpose GPUs (GPGPUs), Field Programmable Gate Arrays(FPGAs), application specific integrated circuits (ASICs), TensorProcessing Units (TPUs)) and system memory, as discussed in greaterdetail below. Implementations also include physical and othercomputer-readable media for carrying or storing computer-executableinstructions and/or data structures. Such computer-readable media can beany available media that can be accessed by a general purpose or specialpurpose computer system. Computer-readable media that storecomputer-executable instructions are computer storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,implementations can comprise at least two distinctly different kinds ofcomputer-readable media: computer storage media (devices) andtransmission media.

Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM,Solid State Drives (“SSDs”) (e.g., RAM-based or Flash-based), ShingledMagnetic Recording (“SMR”) devices, Flash memory, phase-change memory(“PCM”), other types of memory, other optical disk storage, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer.

In one aspect, one or more processors are configured to executeinstructions (e.g., computer-readable instructions, computer-executableinstructions, etc.) to perform any of a plurality of describedoperations. The one or more processors can access information fromsystem memory and/or store information in system memory. The one or moreprocessors can (e.g., automatically) transform information betweendifferent formats, such as, for example, between any of: pipeline data,DevOps data, standardized analytics data, analytics outcomes, log data,normalizer log data, build manifest data, manifests, aggregated logdata, user log data, tool log data, pipeline log data, pipelinemetadata, analytics log data, data patterns, predictive insight models,predictive insights, raw log data, queries, request properties, requestdata types, KPIs, custom dashboards, posture levels, notifications,personas, etc.

System memory can be coupled to the one or more processors and can storeinstructions (e.g., computer-readable instructions, computer-executableinstructions, etc.) executed by the one or more processors. The systemmemory can also be configured to store any of a plurality of other typesof data generated and/or transformed by the described components, suchas, for example, pipeline data, DevOps data, standardized analyticsdata, analytics outcomes, log data, normalizer log data, build manifestdata, manifests, aggregated log data, user log data, tool log data,pipeline log data, pipeline metadata, analytics log data, data patterns,predictive insight models, predictive insights, raw log data, queries,request properties, request data types, KPIs, custom dashboards, posturelevels, notifications, personas, etc.

Implementations of the devices, systems, and methods disclosed hereinmay communicate over a computer network. A “network” is defined as oneor more data links that enable the transport of electronic data betweencomputer systems and/or modules and/or other electronic devices. Wheninformation is transferred or provided over a network or anothercommunications connection (either hardwired, wireless, or a combinationof hardwired or wireless) to a computer, the computer properly views theconnection as a transmission medium. Transmissions media can include anetwork and/or data links which can be used to carry desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer. Combinations of the above should also be includedwithin the scope of computer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to computerstorage media (devices) (or vice versa). For example,computer-executable instructions or data structures received over anetwork or data link can be buffered in RAM within a network interfacemodule (e.g., a “NIC”), and then eventually transferred to computersystem RAM and/or to less volatile computer storage media (devices) at acomputer system. Thus, it should be understood that computer storagemedia (devices) can be included in computer system components that also(or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, in response to execution at a processor, cause a generalpurpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the described aspects maybe practiced in network computing environments with many types ofcomputer system configurations, including, personal computers, desktopcomputers, laptop computers, message processors, hand-held devices,wearable devices, multicore processor systems, multi-processor systems,microprocessor-based or programmable consumer electronics, network PCs,minicomputers, mainframe computers, servers, mobile telephones, PDAs,tablets, routers, switches, and the like. The described aspects may alsobe practiced in distributed system environments where local and remotecomputer systems, which are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network, both perform tasks. In a distributed systemenvironment, program modules may be located in both local and remotememory storage devices.

The described aspects can also be implemented in cloud computingenvironments. In this description and the following claims, “cloudcomputing” is defined as a model for enabling on-demand network accessto a shared pool of configurable computing resources. For example, cloudcomputing can be employed in the marketplace to offer ubiquitous andconvenient on-demand access to the shared pool of configurable computingresources (e.g., compute resources, networking resources, and storageresources). The shared pool of configurable computing resources can beprovisioned via virtualization and released with low effort or serviceprovider interaction, and then scaled accordingly.

A cloud computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. A cloudcomputing model can also expose various service models, such as, forexample, Software as a Service (“SaaS”), Platform as a Service (“PaaS”),and Infrastructure as a Service (“IaaS”). A cloud computing model canalso be deployed using different deployment models such as on premise,private cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the following claims, a “cloudcomputing environment” is an environment in which cloud computing isemployed.

Hybrid cloud deployment models combine portions of other differentdeployment models, such as, for example, a combination of on-premise andpublic, a combination of private and public, a combination of twodifferent public cloud deployment models, etc. Thus, resources utilizedin a hybrid cloud can span different locations, including on-premiseresources, private cloud resources, (e.g., multiple different) publiccloud resources, etc.

At least some embodiments of the disclosure have been directed tocomputer program products comprising such logic (e.g., in the form ofsoftware) stored on any computer useable medium. Such software, whenexecuted in one or more data processing devices, causes a device tooperate as described herein.

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. It will be apparent to persons skilledin the relevant art that various changes in form and detail can be madetherein without departing from the spirit and scope of the disclosure.Thus, the breadth and scope of the present disclosure should not belimited by any of the above-described exemplary embodiments, but shouldbe defined only in accordance with the following claims and theirequivalents. The foregoing description has been presented for thepurposes of illustration and description. It is not intended to beexhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the aboveteaching. Further, it should be noted that any or all of theaforementioned alternate implementations may be used in any combinationdesired to form additional hybrid implementations of the disclosure.

What is claimed:
 1. A method comprising: receiving log data from a plurality of DevOps tools running on computing resources, including: receiving first log data from a first tool running on first computing resources; and receiving second log data from a second tool running on second computing resources; normalizing the received log data, including first log data and the second log data, into a normalized format; storing the received log data in a data lake; accessing a manifest defining log data to be accessed from each of the plurality of tools, including at least the first tool and the second tool; retrieving building manifest data, including the first log data and the second log data, from the data lake responsive to accessing the manifest; aggregating the build manifest data into aggregated log data; and presenting the aggregated log data and presenting the build manifest data in a client DevOps environment.
 2. The method of claim 1, wherein receiving log data from a plurality of DevOps tools comprises receiving log data from a plurality of DevOps tools running on pipeline resources of one or more pipelines, each of the one or more pipelines including at least one pipeline stage.
 3. The method of claim 2, wherein aggregating the first log data and the second log data into aggregated log data comprises sequencing the received log data by pipeline stage of the one or more pipelines into sequenced log data.
 4. The method of claim 3, wherein presenting the aggregated log data in a client DevOps environment comprises presenting the sequenced log data.
 5. The method of claim 1, wherein receiving log data from a plurality of DevOps tools running on pipeline resources comprises receiving log data from a plurality of DevOps tools running on one or more of: cloud resources or on-premise resources.
 6. The method of claim 1, wherein accessing a manifest defining log data to be accessed from each of the plurality of tools comprise accessing a manifest defining log data to be accessed by one or more of: a pipeline identifier, a build identifier, a product, a sprint, or a project.
 7. The method of claim 6, wherein aggregating the first log data and the second log data into aggregated log data comprises indexing the first log data and the second log data by the one or more of: the pipeline identifier, the build identifier, the product, the sprint, or the project.
 8. The method of claim 1, wherein accessing a manifest defining log data to be accessed from each of the plurality of tools comprise accessing a manifest defining log data to be accessed in accordance with one or more filters, the one or more filters defining a log data view granularity.
 9. The method of claim 1, wherein presenting the aggregated log data in a client DevOps environment comprises uploading the aggregated log data to an Information Technology Service Management (ITSM) tool.
 10. The method of claim 1, wherein presenting the aggregated log data in a client DevOps environment comprises sharing the aggregated log data via a collaboration tool.
 11. The method of claim 10, wherein sharing the aggregated log data comprises sharing with an auditor or security requesting one or more of: a review or an approval.
 12. A system comprising: a processor; and system memory coupled to the processor and storing instructions configured to cause the processor to: receive log data from a plurality of DevOps tools running on computing resources, including: receive first log data from a first tool running on first computing resources; and receive second log data from a second tool running on second compute resources; normalize the received log data, including first log data and the second log data, into a normalized format; store the received log data in a data lake; access a manifest defining log data to be accessed from each of the plurality of tools, including at least the first tool and the second tool; retrieve building manifest data, including the first log data and the second log data, from the data lake responsive to accessing the manifest; aggregate build manifest data into aggregated log data; and present the aggregated log data and presenting the build manifest data in a client DevOps environment.
 13. The system of claim 12, wherein instructions configured to receive log data from a plurality of DevOps tools comprise instructions configured to receive log data from a plurality of DevOps tools running on pipeline resources of one or more pipelines, each of the one or more pipelines including at least one pipeline stage.
 14. The system of claim 13, wherein instructions configured to aggregate the first log data and the second log data into aggregated log data comprise instructions configured to sequence the received log data by pipeline stage of the one or more pipelines into sequenced log data.
 15. The system of claim 14, wherein instructions configured to present the aggregated log data in a client DevOps environment comprise instructions configured to present the sequenced log data.
 16. The system of claim 12, wherein instructions configured to receive log data from a plurality of DevOps tools running on pipeline resources comprise instructions configured to receive log data from a plurality of DevOps tools running on one or more of: cloud resources or on-premise resources.
 17. The system of claim 12, wherein instructions configured to access a manifest defining log data to be accessed from each of the plurality of tools comprise instructions configured to access a manifest defining log data to be accessed by one or more of: a pipeline identifier, a build identifier, a product, a sprint, or a project.
 18. The system of claim 17, wherein instructions configured to aggregate the first log data and the second log data into aggregated log data comprise instructions configured to index the first log data and the second log data by the one or more of: the pipeline identifier, the build identifier, the product, the sprint, or the project.
 19. The system of claim 12, wherein instructions configured to access a manifest defining log data to be accessed from each of the plurality of tools comprise instructions configured to access a manifest defining log data to be accessed in accordance with one or more filters, the one or more filters defining a log data view granularity.
 20. The system of claim 12, wherein instructions configured to present the aggregated log data in a client DevOps environment comprises instructions configured to upload the aggregated log data to an Information Technology Service Management (ITSM) tool and share the aggregated log data via a collaboration tool. 